https://www.youtube.com/watch?v=-VEJUzDFAZA

A tough nut to crack

• One of the oldest and hardest problems in computer science

• Major progress from late 2010s onwards thanks to convolutional neural networks

• Mostly solved for English, but not for Arabic

Why so difficult?

• Complex computer vision task
  • Extreme input variability (fonts, layout, degradation, languages)
  • Many ambiguities (e.g., O and 0, l and 1)
• Multimodal problem: Involves combining visual pattern recognition, linguistic knowledge, and world knowledge
  • Humans are incredibly good at this
  • Only with AI/LLMs are we starting to get similar abilities in computers

Why is Arabic OCR not solved?

• Less effort has gone into it
  • English has always been the priority in industry
  • Other languages in Latin script could piggyback on English
  • Less development, less training data for non-Latin scripts
• Script peculiarities
  • Right to left, but often with Latin script mixed in
  • Different positional forms and connections
  • Diacritics
• Currently among hardest languages to OCR
  • Chinese OCR performs better
  • Among big languages, only Urdu OCR performs worse

Adobe Acrobat has no Arabic

Same with most online services

Those that have it perform poorly

Similarly with ABBYY Finereader

Plus: Windows only, and $16+/month

Best GUI solution for now: Google Docs

But important limitations

• No scaling
• No adaptability, modularity

Computer basics

• Computers can be controlled with tools at different levels of abstraction
  • Binary code
  • Low-level languages: Assembly code, C
  • High-level languages: Python, Javascript, R, Unix, Powershell
  • Graphical user interfaces
• All computers have a command line (aka terminal)
  • “Engine room” of computer
  • Language of command line differs by operating system (OS): Unix for MacOS/Linux, Powershell for Windows
• Programming languages like Python and R are OS-independent ways of controlling a computer

Filesystem

• All computers have a filesystem; a tree-structured hierarchy of folders and files
• All folders and files have an “address”, called filepath
• Path notation slightly different on Windows vs MacOS/Linux
  • Unix: /home/thomas/Pictures/photo.jpg
  • Windows: C:\Users\Thomas\Pictures\photo.jpg
• Filepaths can be absolute (in full form) or relative (abbreviated, relative to your location)
• Notion of “location” is important: When coding, you are always launching commands from some place in the filesystem

Programmatic tool types (1): Command line interface (CLI) programs

• Programs that are launched directly in the terminal
• Examples: Tesseract, Kraken, Surya, ImageMagick
• Typically written in C or Python
• Syntax same across operation system (but )
• Many program exists both in CLI form and Python/R package form

Programmatic tool types (2): Python or R packages

• Collections of functions for specialized tasks
• Imported and used inside Python or R
• Examples: pandas in Python or stringr in R
• Names can vary: e.g. Tesseract implemented as pytesseract in Python and tesseract in R

Programmatic tool types (3): Application programming interfaces (APIs)

• Communication channel to a remote service
• E.g. Google Document AI API, OpenAI API
• APIs use HTTP protocol, hence are programming language agnostic
• APIs can be called from command line with curl
• But easier to use in Python or R with tailored packages (e.g. google-cloud-documentai, daiR)

Tool types (4): Local models

• Big files containing weights from AI training, open sourced for public use
• Specialized for some task, e.g. text/image generation, text/image classification
• Used from Python
• In OCR context, we are typically interested in visual large language models, layout parsing models, and large language models (for postprocessing)

GitHub Codespaces - orientation

Git and GitHub

git clone CLONE_URL
# To get the example files etc:
git clone https://github.com/Hegghammer/ocr-cairo

# Make sure you are in the root of the repository
git pull

Object manipulation (R)

# Create scalar
my_text <- "Here is some text"

# Create vectors
texts <- c("Some text", 
           "Some more text", 
           "Even more"
           )
n_words <- c(2, 3, 2)

# Create dataframe
df <- data.frame(texts, n_words)

# Access second element of vector
texts[2]

# Access third row in n_words column
texts$n_words[3]

Object manipulation (Python)

# Create object
my_text = "Here is some text"

# Create lists
texts = ["Some text", "Some more text", "Even more"]
n_words = [2, 3, 2]

# Create dataframe
import pandas as pd
df = pd.DataFrame({
    "Text": texts,
    "Length": n_words
})

# Access second element of list
texts[1]

# Access third row in Length column
df["Length"].iloc[2]

File and folder manipulation (R)

# Create a directory and some files
dir.create("test")
file.create(c("test/notes1.txt", "test/notes2.txt"))

# Get content
contents <- list.files("test")

# Get full paths of content
contents_full <- list.files("test", full.names = TRUE)

# NB: Output of list.files is vector of paths, 
# not files themselves

b. File and folder manipulation (Python)

import os

# Create a directory and some files
os.makedirs("test", exist_ok=True)
open("test/notes1.txt", "w").close()
open("test/notes2.txt", "w").close()

# Get content
contents = os.listdir("test")

# Get full paths
import glob
contents_full = glob.glob("test/*")

# Move file
import shutil
shutil.move("test/notes1.txt", ".")

Iteration (R)

for (SEQUENCE_DEFINITION) {
  INSTRUCTION
}

# Basic loop
for (i in texts) {
  print(i)
}

# Loop over indices
for (i in seq_along(texts)) {
  print(texts[i])
}

# Indices let you do more things
for (i in seq_along(texts)) {
  message("Processing text", i, " of ", length(texts))
  new_name <- paste0("text_", i)
  write(texts[i], new_name)
}

Iteration (Python)

# Basic loop
for t in texts:
    print(t)

# Loop over indices    
for i in range(len(texts)):
    print(texts[i])

# Indices let us do more things
for i, t in enumerate(texts):
    print(f"Processing text {i+1} of {len(texts)}")
    new_name = f"text_{i}.txt"
    with open(new_name, "w") as f:
        f.write(t)

Function building (R)

FUNCTION_NAME <- function() {
  INSTRUCTION
}

# Basic function
say_hello <- function() {
  print("hello!")
}
say_hello()

# One parameter
greet <- function(name) {
  greeting <- paste0("Hello, ", name, "!")
  print(greeting)
}
greet("Rami")

# Two parameters
greet_n <- function(name, n_times) {
  greeting <- paste0("Hello, ", name, "!")
  rep(greeting, n_times)
}
greet_n("Nada", 5)

Function building (Python)

# Basic function
def say_hello():
    print("hello!")

say_hello()

# One param
def greet(name):
    greeting = f"Hello, {name}!"
    print(greeting)

greet("Rami")

# Two params
def greet_n(name, n_times):
    greeting = f"Hello, {name}!"
    return [greeting] * n_times

print(greet_n("Nada", 5))

Text manipulation (R)

library(lorem)
library(readr)
library(stringr)
library(tokenizers)

# Create random content
text <- as.character(ipsum(3))

# Save to file
write(text, "sample.txt")

# Load from file
same_text <- read_file("sample.txt")

# Do things with it
count_words(same_text)
smileys <- str_replace_all(same_text, "\\.", " 🙂")
write(smileys, "smileys.txt")

Text manipulation (Python)

with open(FILENAME, "w") as f:
    f.write(TEXT)
    
with open(FILENAME, "r") as f:
    TEXT = f.read()

import lorem
import re

# Create random content
text = lorem.paragraph() * 3

# Save to file
with open("sample.txt", "w") as f:
    f.write(text)

# Load from file
with open("sample.txt", "r") as f:
    same_text = f.read()

# Do things with it
word_count = len(same_text.split())

smileys = re.sub(r"\.", " 🙂", same_text)

with open("smileys.txt", "w", encoding="utf-8") as f:
    f.write(smileys)

Image manipulation (R)

library(magick)

# Load an example jpeg
files <- list.files("example_docs/columns/orig",
  full.names = TRUE
)
img <- image_read(files[1])
image_info(img)

# Get specific page of a PDF
img2 <- image_read_pdf(files[2], pages = 1)

# Make greyscale
img2_grey <- image_convert(img2, type = "Grayscale")

# crop a 200x150 region starting at (50, 100)
img2_crop <- image_crop(img2, "200x150+50+100")

# Save (use this to convert)
image_write(img2_crop, "test.png", format = "png")

Image manipulation (Python)

import glob
from PIL import Image

# Load an example jpeg
files = glob.glob("example_docs/columns/orig/*")
img = Image.open(files[0])

# Make greyscale
img_grey = img.convert("L")

# Crop a 200x150 region starting at (50, 100)
# PIL crop uses coordinates 
# (left, top, right, bottom)
img_crop = img.crop((50, 100, 250, 250))

# Save (use this to convert)
img_crop.save("test.png", format="PNG")

OCR evaluation (R)

jiwer -r REFERENCE -h HYPOTHESIS

# On command line
pip install jiwer

## WER
command <- "jiwer -g -r sample.txt -h smileys.txt"
wer <- as.numeric(system(command, intern = TRUE))
wer
# Inspect
system("jiwer -g -a -r sample.txt -h smileys.txt")

# CER (add -c)
command <- "jiwer -g -c -r sample.txt -h smileys.txt"
cer <- as.numeric(system(command, intern = TRUE))
cer
# Inspect
system("jiwer -g -a -c -r sample.txt -h smileys.txt")

g. OCR evaluation (Python)

import jiwer

# Read the files
with open("sample.txt", "r") as f:
    ref = f.read()
with open("smileys.txt", "r") as f:
    hyp = f.read()

## WER
wer = jiwer.wer(ref, hyp)
wer

# Inspect
jiwer.process_words(ref, hyp)

# CER
cer = jiwer.cer(ref, hyp)
cer

# Inspect
jiwer.process_characters(ref, hyp)

Three main challenges

• Visual noise (speckles, stains, skew, bleedthrough)
• Unusual script (old typefaces, handwriting)
• Complex layouts (newspapers, magazines, tables)

Character recognition vs layout parsing

• Two very different problems

• Most OCR engines try to do both, but often fail on complex layouts

• Complex layouts often require “the cutout approach”:

  • Identify text blocks
  • Cut them out programmatically
  • OCR them individually
  • Reassemble

Two main strategies: 1) Straight OCR; 2) The cutout approach

Tool types

• Regular engines
  • Local: Tesseract, Kraken, Surya, EasyOCR
  • Remote: Google Document AI
• Visual large language models
  • Local: OLM, Qwen, any LLM with vision
  • Remote: Mistral OCR, ChatGPT, any LLM with vision
• Layout parsers
  • Pretrained: Surya
  • Custom trained: YOLO, LayoutParser

Decision tree

1. How complex is the layout?
   • Low -> Straight OCR
   • High -> Cutout approach
2. Do I mind using a remote service? (for privacy or cost reasons)
   • No -> Use Google Document AI or Mistral OCR
   • Yes -> Use a local engine or VLLM